3D-assisted Facial Texture Super-Resolution

In this paper we propose a new framework for super-resolving facial images under arbitrary pose. While example-based super-resolution methods have demonstrated impressive results for face super-resolution under given pose and imaging conditions, they have limitations dealing with different poses and illuminations. Due to these limitations their application to face super-resolution in generalised situations is either impractical or sub-optimal. In example-based face super-resolution, the quality of the superresolved face depends crucially on how representative the training set is and how well a specific super-resolution method can generalise it and utilise the available information. Due to these issues, example-based methods are limited in handling variations in the subject’s pose or other imaging conditions. Most approaches are either limited to one specific pose (e.g. [1] or [4]) or a number of pre-defined poses for which training data is available (e.g. [3]). A powerful tool introduced by Blanz and Vetter [2] which can describe and synthesise human faces under a large range of poses and imaging conditions is the 3D morphable model. A 3D morphable model is a vector space representation of 3D faces. Given a single 2D face image as input and a set of landmarks, the parameters of the 3D morphable model can be estimated such that they represent the 3D shape and texture of the input face. This process is called model fitting which estimates the model parameters together with a set of rendering parameters such that rendering the model with the estimated parameters will produce an image which resembles the input face image. Model fitting can be formulated as a maximum a posteriori (MAP) estimation of the model and rendering parameters given the input face image and the landmarks. Assuming independence between some parameters: